DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systems

We study the problem of distributed training neural networks (NNs) on devices with heterogeneous, limited, and time-varying availability computational resources. present an adaptive, resource-aware, on-device learning mechanism, DISTREAL, which is able to fully efficiently utilize available resources in a manner, increasing convergence speed. This achieved dropout mechanism that dynamically adjusts complexity NN by randomly dropping filters convolutional layers model. Our main contribution introduction design space exploration (DSE) technique, finds Pareto-optimal per-layer vectors respect resource requirements speed training. Applying this each device select vector fits its without requiring any assistance from server. implement our solution federated (FL) system, where varies both between over time, show through extensive evaluation we are significantly increase state art compromising final accuracy.